On the Dangers of Averaging across Subjects When Using Multidimensional Scaling or the Similarity-choice Model

When ratings of judged similarity or frequencies of stimulus identification are averaged across subjects, the psychological structure ofthe data is fundamentally changed. Regardless of the structure of the individual-subject data, the averaged similarity data will likely be well fit by a standard multidimensional scaling model, and the averaged identification data will likely be well fit by the similarity-choice model. In fact, both models often provide excellent fits to averaged data, even if they fail to fit the data of each individual subject. Thus, a good fit of either model to averaged data cannot be taken as evidence that the model describes the psychological structure that characterizes individual subjects. We hypothesize that these effects are due to the increased symmetry that is a mathematical consequence of the averaging operation. It is common practice to average across subjects when analyzing data. Although this procedure is not without danger (e.g., Estes, 1956; Siegler, 1987), in many cases a good fit of a model to averaged data can be interpreted safely as evidence supporting the basic assumptions underlying the model. Specifically, suppose the data collected from each individual subject reflect a certain type of underlying psychological structure, but also some measurement error. In such a case, we expect the main effect of the averaging operation to be on the measurement error. If the averaging operation affects the underlying structure, then we expect it to work in such a way that the model fits more poorly. For example, suppose learning in some task is all-or-none, but that each subject learns on a different trial. In this case, it is well known that a step function will provide a good fit to the accuracy data of each individual subject, but a poor fit to the averaged data. The worst possible effect of averaging would be to alter the underlying psychological structure of the data in such a way that an invalid model appears valid. For example, the central limit theorem in statistics tells us that if a set of sample means is approximately normally distributed, we learn little about the distribution of the underlying raw data. Rather, normality is a mathematical consequence ofthe averaging operation. As a result, a theory about the distribution of raw data cannot be tested on sample means. This article shows that when ratings of similarity or frequencies of stimulus identifications are averaged across subjects, the psychological structure of the data is fundamentally changed. Specifically, regardless ofthe structure ofthe individual-subject data, the averaged similarity data will likely be fit well by a Address correspondence to F. Gregory Ashby, Department of Psychology, University of California, Santa Barbara, CA 93106; e-mail: [email protected]. standard multidimensional scahng (MDS) model (e.g., Kruskal, 1964a, 1964b; Shepard, 1962a, 1962b; Torgerson, 1958), and the averaged identification data will hkely be fit well by the classical similarity-choice model (SCM; Luce, 1963; Shepard, 1957). We hypothesize that these effects are due to the increased symmetry that is a mathematical consequence of the averaging operation.' MULTIDIMENSIONAL SCALING In MDS, the dissimilarity between a pair of stimuli is assumed to increase with the distance between point representations ofthe stimuli in some psychological space. Typically (i.e., in nonmetric MDS), the observable ratings of judged dissimilarity are assumed to be related only ordinally to the underlying and unobservable perceived dissimilarities. The perceived dissimilarities are assumed to be proportional to psychological distance.^ Let dy denote the perceived dissimilarity between a pair of stimuli / andy, and suppose there are only two relevant psychological dimensions. Let (xi, y,) denote the coordinates of stimulus i in the psychological space. Many different MDS models can be constructed depending on how perceived dissimilarity is computed from the coordinate values of the relevant stimuli. The most widely used class of MDS models assumes dissimilarity is computed from the Minkowski metric: